feat(kernel): forward logs through shared logger - #450

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

Address PR review of the log-forwarding path:

- Drop the cgo.Handle->void* coercion; register with NULL user_data and
  reach the sink via a package global, so no Go pointer is fabricated into
  a C pointer the GC can fault on.
- Forward off the kernel thread: the C callback copies the strings and
  does a non-blocking enqueue onto a bounded channel drained by one
  goroutine, with a drop counter and per-record panic recovery.
- Use an exact-signature C adapter instead of a function-pointer cast.
- Rework logger.sharedOutput into an atomic, LevelWriter-aware proxy: no
  lock held across the user Write (no self-deadlock, a stuck writer stays
  replaceable), nil normalized to io.Discard, and WriteLevel preserved.

Adds enqueue/drop, drain-panic, nil-output, and LevelWriter tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

…og bridge

Follow-up to review feedback on the async log-forwarding path:

- Stamp each kernel record with the emission time captured in the cgo
  callback instead of the drain time. The sink is now a hook-free zerolog
  logger built over the shared output proxy (new logger.Output()), so a
  backed-up drain no longer skews kernel log timestamps.
- Add flushKernelLogs(timeout): a FIFO barrier that drains the async
  hand-off before the writer is closed, retargeted, or the process exits.
  The end-to-end test flushes instead of polling.
- installKernelLogCallback now starts the drain only after a successful
  install and drops the channel (no close) on failure, so there is no
  send-on-closed race to reason about on the error path.
- Surface dropped kernel records with a one-shot warning from the drain
  goroutine (never the kernel thread); the running total stays available
  via kernelLogDropped().
- Document the SetLogOutput concurrent-retarget constraint and add
  concurrent-retarget, nil-discard, LevelWriter, and flush regression tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

…-wrap footgun

Follow-up review fixes:

- HIGH: the one-shot drop warning wrote through the shared logger outside the
  drain's per-record recover, so a panicking writer plus a queue overflow could
  crash the process from the drain goroutine. Both the forwarded record and the
  drop warning now run inside a shared contain() recover. Adds a regression test
  that combines a panicking writer with dropped records.
- MEDIUM: replace logger.Output() (which returned the live proxy, so
  SetLogOutput(Output()) — e.g. a save/restore of the current output — wrapped the
  proxy around itself and deadlocked the next write) with
  logger.NewForwardingLogger(), which returns a hook-free zerolog.Logger over the
  shared output and cannot be round-tripped back into SetLogOutput.
- Clarify the flushKernelLogs doc: it drains the async hand-off for clean
  shutdown/retarget, but has no public entry point yet — only the end-to-end test
  calls it; a supported flush API is a separate change.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

The previous change returned a zerolog.Logger from NewForwardingLogger, but
zerolog.Logger implements io.Writer (value-receiver Write), so
SetLogOutput(NewForwardingLogger()) still compiled and deadlocked: the forwarding
logger's Write routes back through the shared output into the same SyncWriter
mutex. Re-entrant custom writers hit the same trap.

Replace it with logger.ForwardingSink, a method-only type (Event(level)
*zerolog.Event) that is deliberately not an io.Writer, so it cannot be passed to
SetLogOutput to form a self-referential, deadlocking sink. The kernel log sink now
holds a *ForwardingSink and keeps its level mapping. Adds a regression test pinning
that ForwardingSink is not an io.Writer.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

Per-destination serialization holds a non-reentrant lock across the user
Write/WriteLevel, so a destination that logs through the driver from within its own
write (a self-referential sink) deadlocks — and would recurse without bound even if
it did not. This is inherent to any logging library; document the constraint on
SetLogOutput rather than dropping serialization (which would only re-open the
concurrent-corruption case and turn the deadlock into a stack overflow).

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

…ff logger.Logger

Two review fixes:

- Move the pure-Go async forwarding pipeline (record type, bounded queue, drain,
  flush barrier, drop accounting, panic containment) out of the cgo-tagged
  log_callback.go into the untagged logforward_async.go, and its tests into
  logforward_async_test.go. This matches the repo convention documented in
  logging_level.go: the FIFO-flush, drop-policy, and panic-containment tests now run
  in the default CGO_ENABLED=0 lane instead of only the kernel-linked lane.
  log_callback.go keeps only the cgo trampoline and the one-time install.
- Route the drain's one-shot drop warning through the sink's own immutable logger
  (new logSink.warnDropped) instead of logger.Logger, whose embedded value
  SetLogLevel reassigns. The long-lived drain goroutine no longer reads a field that
  races SetLogLevel; drainKernelLogs is now pure Go.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>